Papers with Claude 3.5 Sonnet
Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index (2025.coling-main)
Copied to clipboard
| Challenge: | Prompt engineering is a growing subdiscipline of natural language processing . a lack of appropriate consideration for the financial constraints of computationally burdensome methods can limit their adoption and impact. |
| Approach: | They propose a new metric that combines accuracy scores with token consumption to reflect different resource constraints. |
| Outcome: | The economic prompting index (EPI) measures the performance of 6 prompting techniques across 10 widely-used language models and 4 diverse datasets. |
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs (2025.findings-acl)
Copied to clipboard
Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, Chen Xing
| Challenge: | Existing evaluation frameworks for large language models have limited coverage for multi-turn conversations . multi-turned conversations require accurate instruction following, context allocation, and in-context reasoning at the same time. |
| Approach: | They propose a benchmark to evaluate large language models' ability to conduct multi-turn conversations with humans. |
| Outcome: | The proposed benchmarks achieve near perfect scores on existing benchmarks but only a 41.4% accuracy on the frontier models. |
FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models lack data contamination and complex queries . financial cross-modal multi-hop reasoning is difficult to evaluate and requires precise cross-module reasoning . |
| Approach: | They propose a benchmark to analyze the reasoning capabilities of multimodal large language models. |
| Outcome: | The proposed model is categorized into three difficulty levels—easy, medium, and hard—for step-by-step evaluation. |
Model-Dependent Moderation: Inconsistencies in Hate Speech Detection Across LLM-based Systems (2025.findings-acl)
Copied to clipboard
| Challenge: | Content moderation systems powered by large language models are increasingly deployed to detect hate speech . if two systems produce different outcomes for the same content, it undermines consistency and predictability . |
| Approach: | They analyze 1.3+ million sentences from a factorial design to determine hate speech classification . they find identical content receives markedly different classification values across systems . |
| Outcome: | The proposed model finds that identical content receives markedly different classification values across systems. |
Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game (2024.emnlp-main)
Copied to clipboard
Prisha Samdarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan
| Challenge: | We evaluate the performance of large language models (LLMs) against expert and novice human players. |
| Approach: | They propose to use the New York Times Connections game as a test bed to evaluate the abstract reasoning capabilities of large language models (LLMs) they propose to test the ability of large-language models to be able to cluster and categorize words using semantic relations. |
| Outcome: | The proposed game is a test bed for evaluating abstract reasoning capabilities in humans and AI systems. |